Automatically linking a description of pathology in a medical image report to an image

ABSTRACT

Methods and systems for automatically linking entries in a medical image report to an image of a medical image study. One method includes identifying a description of pathology in text included in the medical image report using natural language processing and applying a model to select for the medical image report at least one modality and at least one image included in the at least one medical image study generated by the at least one modality. The method also includes creating a data link between the description of pathology and the at least one image included in the at least one medical image study generated by the at least one modality, and automatically inserting the data link into the medical image report, the data link being selectable to display the at least one image included in the at least one image generated by the at least one modality.

FIELD

Embodiments described here relate to methods and systems forautomatically linking entries in a medical image report to one or moreimages of a medical image study.

SUMMARY

When viewing a report for a medical image study, a physician viewing thereport (a “reading physician”) may desire to view the images thatcorrespond to a particular finding or statement in the report. However,although the report may be displayed within an image viewer, navigatingto the desired images is a manual process that may take an extendedperiod of time, especially when a large number of images were taken fora particular image study. Furthermore, although some reading physiciansmay include a reference number or identifier for an image that supportsa particular finding or statement, this process is error-prone andrelies on a reading physician to manually add such a reference.Similarly, most reports are plain text (delivered by HL7), whicheliminates the ability to store images in the report and any embeddedlinks actually impede the readability and formatting of the report.Additionally, to be compatible with an image viewer, embedded links useabsolute rather than relative references so that the viewers can belaunched with the full uniform resource locator (URL) context from thereport. In many cases the images and the associated report are sent to adifferent facility for review. Therefore, absolute links are brokensince the image storage system may be inaccessible by the report viewer,which may receive image reports via compact disc (CD) or image sharing.Thus, currently there is no automated means for providing links betweena report and the images the report references.

Accordingly, embodiments described herein provide methods and systemsfor automatically linking entries in a medical image report to an imageof a medical image study. For example, when a report is received, thereport is processed via natural language processing (NLP) to extractimage features (anatomical structures, descriptions of pathology, userartifacts, medical structures, and the like), which can be compared withimage features extracted from one or more images of the image studyassociated with the report. The image features may be extracted from theimages using an anatomical atlas, descriptions of pathology, imagesegmentation, or the like. The image features extracted from the reportmay be based on report sub-headings or from within the body of textincluded in the report. Hyperlinks can then be inserted into the reportwhen an image feature extracted from an image corresponds to an imagefeature extracted from the report. When the inserted hyperlink isselected, the corresponding image including the matching image featurecan be displayed. A similar hyperlink can be inserted (visibly orinvisibly overlaid) on the image, so that a user can select thehyperlink while viewing the image to view the corresponding report and,in particular, the corresponding portion, finding, or statement includedin the report or additionally to the “prior” comparison studies with orwithout the item of interest such as an implanted device or lesion. Insome embodiments, link insertion may be performed by the viewing systemusing an already distributed report, which allows the report links to berelative within the viewing system.

For example, one embodiment provides a method for automatically linkingentries in a medical image report to at least one image of a medicalimage study. The method includes identifying, with at least oneelectronic processor, a first plurality of image features referenced intext included in the medical image report using natural languageprocessing and identifying, with the at least one electronic processor,a second plurality of image features in images included in the medicalimage study. The method also includes comparing, with the at least oneelectronic processor, the first plurality of image features and thesecond plurality of image features. In response to a first image featureincluded in the first plurality of image features and a second imagefeature included in the second plurality of image features matching, themethod includes creating, with the at least one electronic processor, adata link between the medical image report and an image included in themedical image study including the second image feature, andautomatically inserting, with the at least one electronic processor, thedata link into the medical image report, the data link being selectableby a user to display the image.

Another embodiment provides a system for automatically linking entriesin a medical image report to at least one image of a medical imagestudy. The system includes at least one electronic processor. The atleast one electronic processor is configured to identify a firstplurality of image features referenced in text included in the medicalimage report using natural language processing, identify a secondplurality of image features in images included in the medical imagestudy, and compare the first plurality of image features and the secondplurality of image features. In response to a first image featureincluded in the first plurality of image features and a second imagefeature included in the second plurality of image features matching, theat least one electronic processor is further configured to create a datalink between the medical image report and an image included in themedical image study including the second image feature, andautomatically insert the data link into the image, wherein the data linkis selectable to display the medical image report.

Yet another embodiment provides non-transitory, computer-readable mediumstoring instructions that, when executed by at least one electronicprocessor, cause the at least one electronic processor to perform a setof functions. The set of functions includes identifying a firstplurality of annotations referenced in text included in a medical imagereport using natural language processing, identifying a second pluralityof annotations in images included in a medical image study associatedwith the medical image report, and comparing the first plurality ofannotations and the second plurality of annotations. In response to afirst annotation included in the first plurality of annotations and asecond annotation included in the second plurality of annotationsmatching, the set of functions also includes creating a data linkbetween the medical image report an image included in the medical imagestudy including the second annotation, and inserting the data link intothe medical image report, the data link selectable to display the image.

A further embodiment provides a method for automatically linking entriesin a medical image report to an image of at least one medical imagestudy. This method includes identifying, with at least one electronicprocessor, a description of pathology in text included in the medicalimage report using natural language processing and applying, with the atleast one electronic processor, a model to select for the medical imagereport at least one modality and at least one image included in the atleast one medical image study generated by the at least one modality.The method also includes creating, with the at least one electronicprocessor, a data link between the description of pathology and the atleast one image included in the at least one medical image studygenerated by the at least one modality, and automatically inserting,with the at least one electronic processor, the data link into themedical image report, the data link being selectable by a user todisplay the at least one image included in the at least one imagegenerated by the at least one modality.

Similarly, another embodiment provides a system for automaticallylinking entries in a medical image report to an image of at least onemedical image study. The system includes at least one electronicprocessor. The at least one electronic processor is configured toidentify a description of pathology in text included in the medicalimage report using natural language processing and apply a model toselect for the medical image report at least one modality and at leastone image included in the at least one medical image study generated bythe at least one modality. The at least one electronic processor is alsoconfigured to create a data link between the description of pathologyand the at least one image included in the at least one medical imagestudy generated by the at least one modality, and automatically insertthe data link into the medical image report, the data link beingselectable by a user to display the at least one image included in theat least one image generated by the at least one modality.

In addition, an embodiment described herein provides non-transitory,computer-readable medium storing instructions that, when executed by atleast one electronic processor, cause the at least one electronicprocessor to perform a set of functions. The set of functions includesidentifying a description of pathology in text included in a medicalimage report using natural language processing and applying a model toselect for the medical image report at least one modality and at leastone image included in the at least one medical image study generated bythe at least one modality. The set of functions also includes creating adata link between the description of pathology and the at least oneimage included in the at least one medical image study generated by theat least one modality, and automatically inserting the data link intothe medical image report, the data link being selectable by a user todisplay the at least one image included in the at least one imagegenerated by the at least one modality.

Other aspects of the invention will become apparent by consideration ofthe detailed description and accompanying drawings and appendices.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a medical imaging system according to one embodiment.

FIG. 2 illustrates a method of automatically linking entries in adocument to images according to one embodiment.

DETAILED DESCRIPTION

Before any embodiments are explained in detail, it is to be understoodthat the invention is not limited in its application to the details ofconstruction and the arrangement of components set forth in thefollowing description or illustrated in the accompanying drawings. Theinvention is capable of other embodiments and of being practiced or ofbeing carried out in various ways.

Also, it is to be understood that the phraseology and terminology usedherein is for the purpose of description and should not be regarded aslimiting. The use of “including,” “comprising” or “having” andvariations thereof herein is meant to encompass the items listedthereafter and equivalents thereof as well as additional items. Also,electronic communications and notifications may be performed using anyknown means including secure email, direct connections, wiredconnections, wireless connections, etc. It should also be noted that aplurality of hardware- and software-based devices, as well as aplurality of different structural components may be utilized toimplement the invention. Furthermore, and as described in subsequentparagraphs, the specific configurations illustrated in the drawings areintended to exemplify embodiments of the invention, it being understoodthat other alternative configurations are possible.

It should also be noted that embodiments described herein may includehardware, software, and electronic components or modules that, forpurposes of discussion, may be illustrated and described as if themajority of the components were implemented solely in hardware. However,one of ordinary skill in the art, and based on a reading of thisdetailed description, would recognize that, in at least one embodiment,the electronic based aspects of the invention may be implemented insoftware (e.g., stored on non-transitory computer-readable medium)executable by one or more processing units. As such, and by way ofexample, “electronic devices,” “computers,” “computing devices,”“controllers,” “control devices,” or “control modules” described in thepresent specification can include one or more processing units, one ormore computer-readable medium modules, one or more input/outputinterfaces, and various connections (e.g., a system bus) connecting thecomponents.

FIG. 1 illustrates a medical imaging system 100. The system 100 includesa report source 102, an image study source 104, and a user device 106.It should be understood that the system 100 may include additionalcomponents than those illustrated in FIG. 1. For example, the system 100may include a plurality of user devices 106 although only one isillustrated in FIG. 1. Also, in some embodiments, the system 100includes multiple report sources 102, multiple image study sources 104,or a combination thereof. The system 100 may also include one or moreimaging modalities that generate one or more images associated with oneor more types of medical imaging procedures. Furthermore, in someembodiments, the functionality provided by the report source 102 can becombined with the functionality of the image study source 104 in asingle device.

The image study source 104 includes one or more databases that storemedical image studies generated by one or more imaging modalities. Insome embodiments, the image study source 104 includes a picturearchiving and communication system (“PACS”), a cloud storage system, orthe like.

The report source 102 stores image reports. An image report includesfindings by a reading physician based on a set of images (an imagestudy) generated during a medical imaging procedure. In someembodiments, the report source 102 includes a radiology informationsystem (“RIS”). In other embodiments, the report source 102 can includea picture archiving and communication system (“PACS”), an electronicmedical record (“EMR”) system, a dictation system, a cloud storagesystem, a hospital information system (“HIS”), or the like. In someembodiments, the report source 102 is operated by the same entity thatoperates the image study source 104. However, in other embodiments, thereport source 102 and the image study source 104 are operated bydifferent entities. Also, in some embodiments, the image study source104 is included in the report source 102.

The user device 106 is an electronic device operated by an end user,such as a physician, a caretaker, an assistant, a patient, or the like,to view a medical image report. The user device 106 may be aworkstation, a personal computer, a laptop, a tablet computer, asmartphone, a smart watch or other wearable, a smart television, or thelike.

As illustrated in FIG. 1, the user device 106 includes an electronicprocessor 108 a, a computer-readable memory module 108 b, and acommunication interface 108 c. The memory module 108 b can includenon-transitory memory, such as random access memory, read-only memory,or a combination thereof. The electronic processor 108 a can include amicroprocessor configured to execute instructions stored in the memorymodule 108 b. The memory module 108 b can also store data used with andgenerated by execution of the instructions. The communication interface108 c allow the user device 106 to communicate with external devices,including one or more wired or wireless networks 110 (for example, theInternet, a clinical messaging network, a local area network, or thelike) that also communicate with the report source 102, the image studysource 104, or a combination thereof. It should be understood that theuser device 106 may include additional components than those listed inFIG. 1 in various configurations. For example, in some embodiments, theuser device 106 includes a plurality of electronic processors, memorymodules, communication interfaces, or a combination thereof.

As also illustrated in FIG. 1, the memory module 108 b may store animage viewer 112. The image viewer 112, as executed by the electronicprocessor 108 a, allows a user to access stored images and storedreports. In some embodiments, image viewer 112 includes a browserapplication configured to access a web page for accessing the reportsource 102, the image study source 104, or a combination thereof. Inother embodiments, however, the image viewer 112 is a dedicatedapplication for accessing the report source 102, the image study source104, or a combination thereof.

As illustrated in FIG. 1, the report source 102 and the image studysource 104 may include similar components as the user device 106. Forexample, the report source 102 and the image study source 104 may eachinclude one or more servers that each include an electronic processor113 a and 115 a, a computer-readable memory module 113 b and 115 b, andan communication interface 113 c and 115 c similar to the electronicprocessor 108 a, memory module 108 b, and communication interface 108 cdescribed above with respect to the user device 106. Again, it should beunderstood that the report source 102 and the image study source 104 mayinclude additional components than those listed in FIG. 1 in variousconfigurations. For example, in some embodiments, the report source 102,the image study source 104, or a combination thereof includes aplurality of electronic processors, memory modules, communicationinterfaces, or a combination thereof.

As illustrated in FIG. 1, the report source 102 includes (stored withinthe memory module 113 b) a natural language processing (“NLP”) unit 114.In one embodiment, the NLP unit 114 is a software program includinginstructions for processing natural language (textual language) includedin a medical image report. For example, the NLP unit 114 processes textincluded in medical image reports to identify particular keywords,including, for example, image features such as medical features (forexample, organs, organ function or other descriptions of pathology,medical devices or implants, and the like) and user artifacts (forexample, measurements or observations). The extracted features may bemapped to a general ontology such as, for example, UMLS (Unified MedicalLanguage System—Metathesaurus by US National Library of Medicine),SNOMED-CT (Systematized Nomenclature of Medicine—Clinical Terms), aswell as common hierarchies and synonyms for various features of theextracted features. For example, a hip implant may be referred to as thehip prosthesis with sub-components, such as acetabular cup, femoral heador ball, and stem and any of these subcomponents could be key words forthe detected image components.

Similarly, the image study source 104 includes (stored within the memorymodule 115 b) a feature extraction unit 116. In one embodiment, thefeature extraction unit 116 is a software program including instructionsidentifying features in an image, such as anatomical structures orviews, such as Parasternal Long Axis that contains several differentanatomical structures. In some embodiments, the feature extraction unit116 identifies image feature using image segmentation. Imagesegmentation is a process of partitioning a digital image into multiplesegments (sets of pixels) to locate objects and boundaries in images.For example, the feature extraction unit 116 may be configured toprocess an image and determine anatomical localization information usingone or more anatomical atlases or from a different localization orrecognition approach, such as, for example, a neural network recognitionapproach. For example, in a chest computed tomography (“CT”) image,anatomical regions, such as the ascending aorta, left ventricle, T3vertebra, and other regions, can be identified and labeled, such as byusing multi-atlas segmentation. Each atlas includes a medical imageobtained with a known imaging modality and using a particular imagingprocedure and technique wherein each pixel in the medical image islabeled as a particular anatomical structure. Accordingly, an atlas canbe used to label anatomical structures in an image.

After processing an image, the feature extraction unit 116 may output atable of extracted image features. The table lists features (anatomicalstructures, descriptions of pathology, and the like as described above)detected in a set of images or image volume and identifiers of theimages or frames that reference each feature. For example, the liver inan abdominal CT study may be seen in series 3 images 200-240, but for anMR study the liver may be seen in series 1 images 5-10, 20-24, 40-43 asit may be visible in multiple planes of the MR acquisition series.Similarly, in an echocardiography study, the left ventricle (LV) may bepresent in multiple views (2 chamber, 4 chamber, and the like) so the LVmay have 6-10 references in multiple motion images (for example, image 3frames 45-70, image 7 frames 65-80, and the like) and single framecaptures as commonly done for measurements.

It should be understood that the table of features is not limited toanatomical structures. For example, in some embodiments, the featureextraction unit 116 is configured to recognize implanted devices (forexample, artificial hip acetabular implant, vena cava filter, or PICCline) in an image and include references for these devices in the table.The mechanism for the extraction of non-anatomical features may use anentirely disparate mechanism from what is used for the identification ofthe anatomical features. An example of such an object classifier,detector is the IBM Watson Power AI Vision system. However it is alsoadvantageous to use the anatomical location to inform detected devicechoices and speed the matching process. Also, in some embodiments, thefeature extraction unit 116 is configured to extract measurements thathave been performed and added to the images either as part of the pixeldata or as a presentation state reference. For example, with ultrasoundimages it is common for the sonographer to make a number of measurementsduring the exam and these measurements are often saved as image screenshots. These measurements can be further enhanced by using theregistration processes used in the anatomical localization processesdescribed above to match the screen short with the measurement to theframe of the multi-frame acquisition or series (ultrasound image movie).Accordingly, a user could use the table generated by the featureextraction unit 116 to find both the measurement screen shot and theoriginal sequence, which can be displayed to assess the accuracy of themeasurement.

It should be understood that the functionality described herein as beingperformed by the NLP unit 114 and the feature extraction unit 116 may bedistributed and combined in various configurations. For example, in someembodiments, the functionality provided by NLP unit 114, the featureextract unit 116, or both may distributed among multiple modules orsoftware units. Also, in some embodiments, the NLP unit 114 and thefeature extraction unit 116 may both be included in the report source102 or the image study source 104. Furthermore, in some embodiments, theNLP unit 114, the feature extraction unit 116, or both may be includedin the user device 106. Similarly, in some embodiments, the NLP unit114, the feature extraction unit 116, or both are included in one ormore other devices configured to process images and reports as describedherein. For example, in some embodiments, a server may be configured toaccess reports stored in the report source 102 and access images storedin the image study source 104 and perform the feature extraction andnatural language processing as described herein. Additionally, the NLPfeature extraction is not limited to radiology or cardiology reports butis equally applicable to clinical visit notes or surgical notesespecially as the surgical specialists often read their own imagingstudies, refer to image features during a visit or discussion with apatient, or detail the surgical implant or removal of a device all ofwhich can be image correlated.

FIG. 2 illustrates a method 200 of automatically linking entries in areport to images according to one embodiment. The method 200 isdescribed as being performed by the report source 102, the image studysource 104, or a combination thereof. For example, portions of themethod 200 may be performed by the NLP unit 114 and the featureextraction unit 116 (as executed by an electronic processor 113 a, theelectronic processor 115 a, or a combination thereof). However, as notedabove, in some embodiments, all or a portion of the method 200 may beperformed by the user device 106 or other devices with access to reportsand associated images.

As illustrated in FIG. 2, the method 200 includes identifying (forexample, using the electronic processor 113 a of the report source 102),a first plurality of image features referenced in text included in themedical image report using the NLP unit 114 (at block 205). The imagefeatures may include medical structures, user artifacts, or acombination thereof. Medical structures may be anatomical structures(for example, a bone, an organ, a mass or tumor, a ligament, a muscle,and the like), surgical features (for example, an incision, sutures, andthe like), medical devices (for example, a pacemaker, a metal rod orpin, a stent, and the like), or medical implants (for example, a deviceto replace a missing biological structure or descriptions of pathology,such as atrial fibrillation). Medical structures may also includeimaging procedures or parameters, such as “pre-contrast” or“post-contrast.” User artifacts may be measurements, annotations(graphical markings), qualitative values, labels, observations,captions, or other items added to an image by a user (or a computer aspart of an automated analysis of an image). User artifacts may includespecific references such as “suspicious lesion of the liver.” Similarly,a user artifact may include a reference to a particular key image orpresentation state.

The NLP unit 114 may be configured to store data regarding theidentified first plurality of image features in a data structure such asa table. For example, when a physician prepares a medical image report,the physician uses terms to describe image features, including medicalstructures and user artifacts. The electronic processor 113 a, using theNLP unit 114, processes text inserted into a report to detect suchterms. The NLP unit 114 may be configured to store these detected terms(referencing image features) in the table along with data identifyingone or more locations within the report including the detected term (orassociated terms). The locations may include page numbers, line numbers,section numbers, or the like. The NLP unit 114 may be configured processtext included in a body of the report. However, alternatively to inaddition, the NLP unit 114 may be configured to use a format orstructure of a report to identify the first plurality of medicalstructures. For example, the medical image report may process reportsubsections or headings for individual body parts, anatomical systems,types of images, and the like. The electronic processor 113 a, using theNLP unit 114, may utilize the subsection names to quickly identifyrelevant terms used in the subsection. For example, the NLP unit 114 maydetermine that a subsection is named “Spine” and will look for languagerelated to the spine in the subsection, such as “vertebra” or a specificstructure or grouping (e.g. L5-S1 or sacrum, which is composed ofmultiple vertebrae or sub feature such as the right transverse processor spinous process of L3).

For some anatomical locations and devices there are a number of synonymsor acronyms (for example, “RCA” vs “Right Coronary Artery”) and the NLPunit 114 may be configured to resolve these synonyms and acronyms andmap these terms to supported anatomical locations or devices (locationsor devices identified by the feature extraction unit 116 as describedbelow). Additionally there may be greater or lesser specificity in thetext than is supported in the anatomical localization performed by thefeature extraction unit 116. For example, “Aortic Arch” may be mentionedin a report but the feature extraction unit 116 may be configured toperform anatomical localization to identify “ascending aorta” and“descending aorta” but not “aortic arch” or “aorta”. Additionally, inthe case of devices the feature processer may be able to recognizesub-components (e.g., acetabular cup vs femoral head vs stem) in a hipimplant prosthesis. Accordingly, the NLP unit 114 may be configured tocompare and map identified image features from a report to knownfeatures that may be extracted by the feature extraction unit 116. Insome embodiments, to apply finer gradations, the NLP unit 114 may addadditional layers of subdivisions to the atlas segmentation orsubsequently apply additional rules to enable finer subdivisions (forexample, proximal, mid, distal segments of bone or vessels, medial orlateral aspect/side of a structure, or superior vs inferior aspect).

As illustrated in FIG. 2, the method 200 also includes identifying (forexample, with the electronic processor 115 a) a second plurality ofimage features in images included in an image study associated with themedical image report via the feature extraction unit 116 (at block 210).The image study may be associated with the medical image by a uniqueidentifier of the image study included in the image report. In otherembodiments, information included in the report, such as patient,procedure date, and the like, may be used to identify the associatedimage study. The feature extraction unit 116 may be configured toanalyze pixels in one or more images included in the image study toidentify the second plurality of image features structures. For example,as described above, the feature extraction unit 116 may be configured touse image segmentation to identify anatomical structures in an image.The feature extraction unit 116 may also be configured to processmetadata (such as header information) of an image to identify an imagefeature, such as user artifacts. Similar to the NLP unit 114, thefeature extraction unit 116 may be configured to store identified imagefeatures in a data structure, such as a table. The table may include theidentified features and a reference to a particular image, frame, orportion including the identified feature.

The method 200 also includes comparing the first plurality of imagefeatures and the second plurality of image features (at block 215) todetermine if any features in the first plurality of image features andthe second plurality of image features match. For example, the firstplurality of image features may include a medical structure identifiedas a “liver,” and the second plurality of image features may include amedical structure identified as a “liver.” It should be understood thatidentified image features may “match” when they are identical orrelated. For example, when the first plurality of image featuresincludes a “marked tumor” and the second plurality of image featuresinclude a graphical annotation marking a portion of an image, thesefeatures may be considered “matching.” Similarly, when the firstplurality of image features includes a “cranium” and the secondplurality of image features includes a “skull,” these features may beconsidered “matching” even though they are not identical. The use of anontology or other hierarchical organization where the featuresidentified in the report and the features identified in the imagesenables the resolution of differing levels of specificity found in thetext or images. This is especially notable when a sub structure ofeither the anatomy (e.g. vertebral facet) or device (acetabular cup) orspecific sub structures of the brain which are finer in granularity thanmay be supported in the anatomical localization or segmentation ismentioned in the report. Furthermore, additional location modifiers maybe used with a given structure to add finer location information such asthe distal portion of the femur, medial portion of the right side of themandible can provide hints as to the best slices or views to use where aportion of the anatomy covers multiple images or slices. The modifiersdiscovered in the report are not limited to physical location but mayalso include other characteristics. For example, an “enhancing” lesionindicates that this was seen in the series that was taken after theaddition of a contrast medium (e.g., iodinated contrast for CT,gadolinium for MR or other) and in the case of a study with bothnon-contrast and contrast series, the contrast series may be linked tothe “enhancing” lesion mention in the report rather than thenon-contrast series or study.

In response to the first plurality of image features and the secondplurality of image features not including any matching features (“No” atblock 220), the method 200 may consider other images included in imagestudy or, if all of the images have been processed, return to waitingfor a new medical image report and a new medical image to identifyfeatures within (such as at blocks 205 and 210, respectively).

Alternatively, in response to the first plurality of image features andthe second plurality of image features including a matching feature(“Yes” at block 220), a data link is created between the medical imagereport and at least one image (at block 225) and is inserted into themedical image report (at block 230). The data link may be inserted at alocation of the matching image feature, such as a matching medicalstructure. For example, if both the report and an image in theassociated image study reference a “liver,” a data link to an image andthe structure or coordinates illustrating a liver may be inserted intothe report, such as in a subsection of the report relating to the liver.The insertion of these data links may be especially useful to a layperson or referring physician who may not be adept at discerning thevarious structures especially in cross-sectional anatomy, such as, forexample, CT or MR. For example, in the above example of a “liver,” theright image or image view may be selected but optionally a highlight ofthe liver, such as an outline around the liver (displayed statically ormomentary, such as displayed for three seconds and then disappeared). Insome embodiments, the anatomy may also be shown in multiple viewswhether these views were present in the original study. For example,with reference to L3 a user may be interested in viewing both the axialview and the sagittal view simultaneously.

The data link (for example, a uniform resource locator (“URL”))identifies the medical image associated with the medical image report.Accordingly, the data link is selectable (for example, by a user of theuser device 106 clicking on the data link) to view the medical image.The data link may be inserted in the medical report approximate to thematching feature. Thus, as a user reviews the report, the user canquickly access an image supporting or associated with a particularfinding or statement in the report. Thus, by creating the data link whenthe medical image report and the at least one medical image contain thesame image feature (for example, medical structure), the user canquickly select the data link to access and view the medical image (orportion of the medical image), which improves workflow, prevent errors,and limits wasted computer resources associated with manually attemptingto identify a relevant image to a portion of a report.

The data link may be inserted in to the report as a standalone link.However, in other embodiments, the data link may be inserted or added toexisting text in the report. For example, if a mass is described in asubsection of the medical image report, the electronic processor 108 amay select the specific sentence or subsection that describes the massand insert the data link into the sentence or subsection (by adding thedata link as selectable text or by creating a hyperlink over existingtext in the medical image report). Also, in some embodiments, thehyperlinks may be summarized in a table inserted into the report, whichserves as a summary of the images linked with the report. Accordingly, auser can select a data link by clicking on a heading, a text highlightlink, or entries in a table of references and the image viewer 112automatically navigates to the corresponding image. In some embodiments,an optional overlay is also provided on the image (for example, ananatomical localization overlay, an implant overlay, a measurementoverlay, or the like) that outlines, shades, labels, or otherwisehighlight an image feature of the image, such as an anatomicalstructure, measurement, or implanted device. Also, in some embodiments,the matched image feature (or a portion thereof) from the report may beoverlaid on the image. For example, the textual language included in afinding or a measurement may be displayed with the linked image so thata user does not need to flip back and forth between the report and theimage. For example, information from both the body of the report (forexample, the findings section) and the impressions could be displayedtogether on an image. Combining this data allows a user to see anyrecommendations that may have been made based on the findings made froma particular image. Also, in some embodiments, default text or data maybe added to an image associated with a data link (regardless of whetherthe text or data is included in the associated report). For example,relevant measurement criteria may be added to an image. As one example,if lung nodule has been identified in an image, displaying theFleischner criteria may be added to the image to improve a user'sconfidence in follow-up recommended in the report (or modify therecommendation as needed). Thus, as noted above, the data links, andtheir automatic creation, simplifies the navigation of images by a userand quickly and accurately shows a user what images and particular imagefeatures are being discussed in the report in an unambiguous manner.

For example, the data links may be used to link measurements in a reportto the original images from which the measurements were derived. Asmentioned previously, it is common in many imaging studies to makemeasurements during the procedure. These measurements are typicallysaved as part of the image record. The measurements are also commonlyimported in to the reports (manually or automatically whether via DICOMSR, OCR, or the like). However, in the import process, the linkage tothe image is not preserved so the user has a list or table ofmeasurements but then the user must look through the images to find eachmeasurement. This is a common task for a cardiologist who is reading anechocardiography study and wants to check the measurements taken by thesonographer for accuracy. The measurements would be extracted by thefeature extraction unit 116 (via OCR) from images (screen shots) andincluded in the table of extracted features table. Accordingly, a usercan click on a measurement (or its label) and navigate directly to therelevant images including both the still frame or secondary capture withthe measurement and the source multiframe sequence or series from whichit was derived so as to ascertain if this is the “best image” andaccurate measurement or if they may like to add more measurements asmight be done during the creation of the report. As described in moredetail below, the automatic linkage described herein may be provided tolink to a current imaging study and optionally to prior comparisonsincluding, for example, the identification of optimum comparison studiesand images within them.

Similarly, the data links described above may be used to location imagesassociated with qualitative text. For example, by clicking onqualitative text included in a report, the user is automatically takento the images associated with the text and may highlight themeasurements or values associated with the qualitative text.

In some embodiments, if the medical image report is not a document typethat supports data links, the medical image report may be converted to adocument type that does support data links. For example, the medicalimage report may be converted to an HTML document, RTF document, PDFdocument, or the like, which supports data links. In some embodiments, auser may review generated data links and approve or reject links. Thisuser feedback may be used, with one or more machine learning techniques,to improve the natural language processing, features extraction,comparing, generating and insertion of data links, or a combinationthereof.

In some embodiments, the data links are used as a lookup into the tablecontaining the image features extracted from an image. Accordingly, whenthe table containing the extracted image features are stored in a staticlocation, the data links may be preserved even when the report and theimages are stored separately from the report (such as in an inaccessiblestorage location) or subsequently moved to a new location. Also, in someembodiments, the report can be provided with the table of extractedimage features so that when the images or the report are exported (forexample, on CD, DVD, USB, or the like) that data links between thereport and the images are still available and able to be viewed withinthe typically included viewer in a linked manner or at least navigableby text references in the table (for example, L3 was found in Series 2images 34-37). Alternatively, in the post processing of the report,thumbnail images with links can be inserted rather than merely a texttable.

As described above, in response to a user selecting the data link (suchas through the image viewer 112 executed by the user device 106), thelinked medical image is displayed to the user. In some embodiments, theimage viewer 112 is configured to display both medical image reports aswell as images. However, in other embodiments, a separate window(separate from a window displaying the report) may be generated todisplay the linked medical image. Also, in some embodiments as describedabove, a feature identified in an image is highlighted when displayedthrough selection of a data link. For example, an image feature may behighlighted, flashed, labeled, or otherwise marked within an image tofurther focus the user's attention on the relevant section of the image.Furthermore, selecting a data link may display a plurality of images.For example, an image study may contain multiple views of a medicalstructure, and, when the electronic processor 108 a creates the datalink, the data link may reference all the relevant images. When the datalink is selected, all of the relevant images may be displayed and theuser may scroll or otherwise navigate through the images. Alternativelyor in addition, a list of the relevant images may be initially displayedwhen a user selects the data link and a user can select a relevant imagefrom the list for display. Also, in some embodiments, multiple imagesmay be displayed simultaneously, such as to provide a side-by-sidecomparison. Such a comparison may be automatically generated orgenerated by a user selecting images for inclusion in the comparison.Similarly, in some embodiments, one or more rules may be applied toimages associated with a selected data link to determine or prioritizethe images for display to a user. For example, the rules may definewhich images or portions of images (or which portions of the medicalimage report) are more or less relevant to an identified image feature.For example, when a medical image report contains the phrase “tumorshown in sagittal view of brain,” the rules may specify that sagittalviews of the brain may be displayed before other views of the brain. Therules may be set by a user or may be automatically learned using one ormore machine learning techniques. For example, if a reading physicianfrequently uses a certain view of an organ when reviewing medical imagereports, a rule may be generated that prioritizes this view whencreating a data link.

It should be understood that the data link may, alternatively or inaddition, be inserted into the medical image. Accordingly, when a userviews the medical image, the user may be able to select the data link toautomatically access the report associated with the medical image, and,in particular, the particular portion of the report referencing themedical image (or an image feature contained therein). Thus, a data linkmay be used to create a two-way link between a report (a section of areport) and an associated image. As described above for the data linkinserted in the medical image report, when a user selects a data linkinserted in a medical image, the report may be displayed and a portionof the report may be highlighted to draw the user's attention to therelevant section of the report. When the data link is selected, theelectronic processor 108 a is configured to display the medical imagereport (at block 335). For example, when a circled mass is selected, theelectronic processor 108 a displays portions of the medical image reportthat reference the mass. In some embodiments, the electronic processor108 a highlights the portions of the medical image report that arerelevant to the image feature that was selected.

It should be understood that the method 200 described above may beconfigured to set the type of image features identified in reports andassociated images. For example, the method 200 may be configured to onlyidentify medical structures, only identify particular types of userartifacts, or a combination thereof. Also, the method 200 may beperformed or triggered in response to various events. For example, insome embodiments, the report and associated images may be processed asdescribed above in response to a receiving a request from a user to viewa report or an image study. Alternatively, due to the computational loadof this process, a report and associated images may be processed asdescribed above when the report is stored to the report source 102.Similarly, reports and associated images may be pre-processed accordingto a predetermined schedule whether for a current study or somerules-based selection of prior comparison studies or based on atriggering event such as an admission of a patient, ordering of a followup study, or scheduling of an appointment or ingestion of a new imagingstudy for that patient.

In some embodiments, the electronic processor 115 a may access furtherimage studies or other data associated with a patient as part ofprocessing a report for the patient. For example, the electronicprocessor 115 a may access image studies done prior to a current imagestudy to determine if a medical structure or image feature can be foundin the prior image studies. Finding relevant medical structures or imagefeatures in prior image studies and creating data links to the priorimages studies may assist the reading physician in analyzing the medicalimage report.

For example, since patient demographics are included in a report, thisdata can be used to link to patient image studies stored in the imagestudy source and a data link may be added to the demographics in thereport that allows a user to quickly access these additional studies.Similarly, a data link may be added to a date in the report and a usercan select the data links to see all image studies performed on thatdate (for the patient in question or all patients, for the same imagingmodality or all modalities, for the same imaging clinic or all clinicsin a network, or a combination thereof). This type of data link may beuseful when a patient has a CT head scan and a CT cervical spine scanperformed on the same date but they are dictated as separate studies.Similarly, if a report references a follow-up imaging procedure, a datalink could be added to this reference that links to the image studygenerated as part of this recommended follow-up procedure. In addition,if a follow-up image study is not located, one or more alerts may begenerated indicating that the follow-up may not have been performed.Accordingly, the linking between image studies may provide uniquetracing of recommendations and the follow-ups which could originateeither in the currently viewed report or anywhere in the series ofreports that constitute the longitudinal record for a patient.

Data links could also be provided to one or more images of prior imagingstudies, including when referenced in a “comparisons” section of areport or the body of the report. For example, when a reading physiciandictates “On today's exam the mass measures x by y by z, while on theprior study this measured a by b by c,” a data link may be added to bothmeasurements that, when selected, display the relevant images used forthe measurements. In some embodiments, the relevant images for bothmeasurements may also be displayed for comparison purposes, such as in aside-by-side orientation. Accordingly, the data links described hereinmay be generated not just to access images within a single image studybut provides depth in the longitudinal record (such as for studiesoccurring within a predetermined window) and the ability to locate arelevant prior image study, open the study for comparison and navigateto the images or other objects, such as the Pressure waveforms or ECG,that are outside of the imaging study that the current report is for. Asnoted above, the extracted image features are not limited to anatomicalstructures but may also include descriptions of pathology (disease stateor functional pathology). This type of NLP identification and imagenavigation may require more complex rule sets. For example, a proximalright coronary artery stenosis has a one-to-one text to imagerelationship. However, some disease states may have a one-to-many textto image relationship or navigation that requires more complex rules.For example, to generate disease-state (pathology) linkages, the systemsand methods described above may use a top-level hierarchy that performsa rule-based determination of a simple vs a complex model. The simplemodel may involve a single modality (for example, an echo US) and asimple disease state. For example, when the report text includes thephrase “Aortic Stenosis,” the systems and methods described above mayselect a modality, an image, and an image filter for measurements,wherein the modality associated with the report from which the text wasretrieve is selected, the images to display include the ParasternalShort Axis (PSAX) image, which provides a best view to visualize valveanatomy, the Parasternal Long Axis (PLAX) image, which provides a bestview to visualize valve motion, and a Continuous wave (CW) Doppler imageof the aortic valve, which is best to evaluate function. With theseproper image selections, the measurements associated with these selectedimages are considered the relevant measurements. Accordingly, with theseselections, when a user selects “Aortic Value Mean Gradient” measurementin the report, the CW Doppler image is displayed. Similarly, if the userselects the “Aortic Valve Peak Velocity” measurement in the report, thePLAX and CW Doppler images are displayed.

In contrast, the complex model may handle multiple modalities andsituations where the disease state is not linked to solely onestructure. In this model, additional levels of organization may beemployed. Also, due to the number of images that may be associated withthe disease state, a priority order may be employed. For example, thecomplex model may be used to select a primary modality (modalityassociated with the report from which the text is selected), a primarymodality image selection, a primary Image filter for measurements, and apriority ordering of images. The model may also make similar selectionsfor a secondary modality selection, a third modality selection, and thelike. For example, when a report includes the text “Hypertrophiccardiomyopathy (HCM),” the complex model may select the following echoimages (as the primary modality) in the following order to display:Parasternal long axis (PLAX) with 2D measurements, Parasternal shortaxis (PSAX) with 2D measurements, Left ventricle strain rate curve,Mitral valve Doppler with measurements. Tissue Doppler withmeasurements, Pulmonary vein Doppler with measurements, and ContinuousWave Doppler (CW) of tricuspid valve with measurements. Similarly, thecomplex model may select the following cardiac MR images (as thesecondary modality) in the following order to display: four chamber enddiastolic view with and without contrast.

It should be understood that although the previous examples select ameasurement to associate with the data link, such a measurement isoptional. For example, some disease states may not be associated with aspecific measurement although they may be associated with a particularanatomical structure.

Thus, embodiments of the invention provide methods and systems forautomatically linking entries in a medical image report to an image. Itshould be understood that while embodiments described herein aredirected towards medical image reports, the methods and systems may beapplied to other types of documents, including, for example, operativereports and clinical procedure notes. Similarly, videos, audiorecordings, or a combination thereof may be processed to extract imagefeatures that can be linked to a corresponding report or image asdescribed above.

Various features and advantages of the invention are set forth in thefollowing claims.

What is claimed is:
 1. A method for automatically linking entries in amedical image report to an image of at least one medical image study,the method comprising: identifying, with at least one electronicprocessor, a description of pathology in text included in the medicalimage report using natural language processing; applying, with the atleast one electronic processor, a model to select for the medical imagereport at least one modality and at least one image included in the atleast one medical image study generated by the at least one modality;creating, with the at least one electronic processor, a data linkbetween the description of pathology and the at least one image includedin the at least one medical image study generated by the at least onemodality; and automatically inserting, with the at least one electronicprocessor, the data link into the medical image report, the data linkbeing selectable by a user to display the at least one image included inthe at least one image generated by the at least one modality.
 2. Themethod of claim 1, wherein applying the model further includes applyingthe model to select at least one image measurement included in the atleast one medical image study and wherein creating the data linkincludes creating a data link between the description of pathology, theat least one image included in the at least one medical image studygenerated by the at least one modality, and the at least one imagemeasurement included in the at least one medical image study.
 3. Themethod of claim 1, further comprising selecting the model from aplurality of models, wherein at least one of the plurality of models isassociated with a description of pathology associated with multipleimaging modalities.
 4. The method of claim 1, further comprisingselecting the model from a plurality of models, wherein at least one ofthe plurality of models is associated with a description of pathologyassociated with multiple anatomical structures.
 5. The method of claim1, wherein the at least one image includes a plurality of images andfurther comprising applying the model to order the plurality of imagesfor display in response to selection of the data link.
 6. The method ofclaim 1, wherein the at least one modality includes a primary modalityand a secondary modality and wherein applying the model to select the atleast one image includes selecting at least one image generated by theprimary modality and at least one image generated by the secondarymodality.
 7. The method of claim 1, further comprising identifying atleast one image feature in the at least one image associated with thedescription of pathology and wherein creating the data link includescreating the data link between the description of pathology at the atleast one image feature.
 8. The method of claim 1, further comprisingfurther comprising generating a table including the at least one imageand wherein the data link is used as a lookup into the table.
 9. Themethod of claim 1, further comprising generating an overlay for the atleast one image based on the data link, wherein the overlay highlightsat least one image feature in the at least one image associated with thedescription of pathology.
 10. The method of claim 1, further comprisinggenerating an overlay for the at least one image based on the data link,wherein the overlay includes text based on the description of pathology.11. The method of claim 1, wherein the medical image report isassociated with a first medical image study and wherein applying themodel to select the at least one image included in the at least onemedical image study includes applying the model to select the at leastone image from a second medical image study, the second medical imagestudy being a prior image study for a patient associated with the firstmedical image study.
 12. A system for automatically linking entries in amedical image report to an image of at least one medical image study,the system comprising: at least one electronic processor configured toidentify a description of pathology in text included in the medicalimage report using natural language processing, apply a model to selectfor the medical image report at least one modality and at least oneimage included in the at least one medical image study generated by theat least one modality, create a data link between the description ofpathology and the at least one image included in the at least onemedical image study generated by the at least one modality, andautomatically insert the data link into the medical image report, thedata link being selectable by a user to display the at least one imageincluded in the at least one image generated by the at least onemodality.
 13. The system of claim 12, wherein the at least oneelectronic processor is further configured to apply the model to selectat least one image measurement included in the at least one medicalimage study and wherein the data link is between the description ofpathology, the at least one image included in the at least one medicalimage study generated by the at least one modality, and the at least oneimage measurement included in the at least one medical image study. 14.The system of claim 12, wherein the at least one image includes aplurality of images and wherein the at least one electronic processor isconfigured to apply the model to order the plurality of images fordisplay in response to selection of the data link.
 15. The system ofclaim 12, wherein the at least one modality includes a primary modalityand a secondary modality and wherein the at least one electronicprocessor is configured to apply the model to select the at least oneimage by selecting at least one image generated by the primary modalityand at least one image generated by the secondary modality.
 16. Thesystem of claim 12, wherein the medical image report is associated witha first medical image study and wherein the at least one medical imagestudy includes a second medical image study, the second medical imagestudy being a prior image study for a patient associated with the firstmedical image study.
 17. Non-transitory, computer-readable mediumstoring instructions that, when executed by at least one electronicprocessor, cause the at least one electronic processor to perform a setof functions, the set of functions comprising: identifying a descriptionof pathology in text included in a medical image report using naturallanguage processing; applying a model to select for the medical imagereport at least one modality and at least one image included in the atleast one medical image study generated by the at least one modality,creating a data link between the description of pathology and the atleast one image included in the at least one medical image studygenerated by the at least one modality; and automatically inserting thedata link into the medical image report, the data link being selectableby a user to display the at least one image included in the at least oneimage generated by the at least one modality.
 18. The non-transitory,computer-readable medium of claim 17, wherein the set of functionsfurther comprising applying the model to select at least one imagemeasurement included in the at least one medical image study and whereinthe data link is between the description of pathology, the at least oneimage included in the at least one medical image study generated by theat least one modality, and the at least one image measurement includedin the at least one medical image study.
 19. The non-transitory,computer-readable medium of claim 17, wherein the at least one imageincludes a plurality of images and wherein applying the model includesapplying the model to order the plurality of images for display inresponse to selection of the data link.
 20. The non-transitory,computer-readable medium of claim 17, wherein the at least one modalityincludes a primary modality and a secondary modality and whereinapplying the model includes applying the model to select the at leastone image includes selecting at least one image generated by the primarymodality and at least one image generated by the secondary modality.